Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setup build automation script and CI workflow #2

Closed
wants to merge 67 commits into from

Conversation

sudotensor
Copy link
Contributor

Build Automation Script

Currently, to setup the two examples (MCUBoot and Mock) for demonstration, the build instructions in their associated documentation has to be followed step-by-step. This process can be quite cumbersome and isn't entirely ideal for testing.

Instead, this build process can be automated through a simple bash script that performs every step in the build instructions for the end-user - as a result, the fota.sh cli build tool was created. The tool takes in as arguments the example, target board, mount point of target board, and toolchain.

Note: If the mount point is not provided (it could be the case that an end-user is trying to build without the board connected), then the target binary is not flashed. This is especially useful for building the examples with a CI workflow.

In the case of the MCUboot example, the "factory firmware" is saved and would require manual transfer when the board is indeed connected. In either case, the demonstration step would be the only part that requires manual intervention from the user.

Important: For now, the tool assumes the target board, toolchain, and example unless otherwise specified by the end-user. Currently, the only board and toolchain supported are NRF52840_DK and GCC_ARM respectively.

There are slight kludges with how the MCUboot and Mock examples are built. The former is a direct result of dependency conflicts between mbed-tools, pyocd, and mbed-os. The latter is a result of issues with flashing the target binary to the connected board. The details of both are addressed below:

  1. Dependency Conflicts: The difference in version requirements of the PyYAML dependency imposed by both mbed-os and pyocd led to the creation of a temporary virtual environment; this is used in the "creating and flashing the factory firmware" stage of the MCUboot example build process. This is a known issue and would require changes to the requirements.txt file in the mbed-os to resolve.

    Another minor conflict that doesn't hinder the build process is one between mbed-tools and mbed-os on the version requirement of the Click dependency; mbed-tools requires that the minimum version of Click to be greater than 7.1 while mbed-os requires it to be greater than 7.0. Now, the dependencies for mbed-os are installed after those of mbed-tools, which overwrites the newer version and generates a conflict. This would (again) require changes to the requirements file in the mbed-os repository to bump up the minimum version number of Click to 7.1.

  2. Target Binary Flashing: For the Mock example, compiling with mbed-tools results in the following error:

    ERROR: Build program file (firmware) not found <path to mock example>/target/cmake_build/<target board>/develop/<toolchain>/target.hex
    

    The tool is looking for a hex file under the cmake build output whose name comes from project directory (in this case, "target"). However, the generated one is named BLE_GattServer_FOTAService.hex. Again, this is a known issue and has been filed (282) in the mbed-tools repository by noonfom.

GitHub CI Workflow

Currently, there is no CI workflow setup to make sure that both examples build successfully. The aim was to setup a GitHub actions workflow that uses the new CLI build tool fota.sh (from branch build-process-automation) to build the examples and check for any errors; known issues (documented above) are ignored or have workarounds implemented to avoid them.

A GitHub workflow "Build Examples" was created to build the two examples (as two separate jobs). Also, this workflow uses the mbed-os-env container as it bundles the GCC ARM toolchain, along with Python and CMake.

Note: There was a slight hiccup in setting up the workflow that has been documented in a comment here.

Remarks

The description of this pull request was adapted from the issues and pull requests made in my fork.

For the automation script, the aim was to write it in bash as it is widely available and easily maintainable. In the future, the script's functionality could be migrated into a CLI tool written in Python.

It was not a strict requirement to run the examples on the target board using the CI workflow by means of a Raspberry Pi rig connected to an NRF52840_DK target board. However, support for this can be added later considering that the new build tool supports building the examples without providing a mount point.

The script is setup to take in, as its first parameter, the name of the
example to build. This is then checked against the valid examples and
matched with the associated directory. The directory is then checked for
in the current path - if not, it's an indication that the script was not
sourced from the project root.
The script now sets up the virtual environment directory and installs
the required python dependencies from requirements.txt.

Note: This script may have compatibility issues with Windows that will
be addressed in a separate issue.
This commit saw a few major changes, including a correction to the
target board variable. The mount point is now passed in as an argument
to the script as there's an issue with flashing with mbed-tools; please
refer to issue #282 on the mbed-tools repository for more information.
As a result, the binary is flashed manually.
The script has many caveats that are addressed as a comment on the
associated issue (or pull request). These are mainly as a result of
dependency conflicts between mbed-tools, mbed-os, and pyocd. The updated
script follows the build instructions as listed in the documentation
accompanying this example (with additional checks).
This commit is the first in a series of refactoring commits. It
introduces a new tool, fota.sh. This commit sets up the script with an
author's note and a function to display the tool's usage.
The options passed into the script are parsed and stored in the
corresponding variables. Note that some of these variables have default
values (as indicated by the usage message).
This commit also removes the print_options function, which was created
to help debug the functionality of the parse_options function.
The file is setup with constants for text formatting.
The say function prints a message to the console with formatting based
on the selected formatting mode. For now, the error message
functionality of this function has been implemented. Refer to the
comments in the script for more information.
The function prints an error message to stderr and exits or returns with
a code of 1 based on the fail mode, which is passed as an input. If the
file is being sourced, then "return" would be passed in; else "exit"
would be passed in. This function invokes the say function created
earlier with the error formatting mode.
Previously, the parse_options function required that the option be
followed by an '=' symbol and the value (i.e -o=value or
--option=value). The usage instructions separate the option and the
value by a space and the function was modified to meet this requirement.

The following stack overflow post proved quite useful in understanding
how options are parsed - in fact, the "Bash Equals-Separated" part of
the accepted answer was adapted for the intial implementation.

https://stackoverflow.com/a/14203146
The parsing function would loop indefinintely if there was no value
associated with an option. There is still a need to check if the default
value of the options have been cleared, in which case the prior default
must be restored or an error must be displayed indicating that the
option was not followed by a value.
This commit rolls back the parsing function to it's state in commit
a58f3b9. The equals-separated parsing
seems a bit more robust than space-separated parsing. I did experiment
with getopt, which didn't work as the version pre-installed on macOS
doesn't support long option parsing. The 'enhanced' getopt would require
a separate installation - at this phase, a python script seems easier.
The fota script now sources the utils script and uses the fail and
setup_formatting functions to handle the error of unrecognised options
in the parsing phase.
A note is displayed if no mount point is specfied indicating that the
binaries will have to be flashed manually.
The message is now updated to use the note formatting and the second
line was dropped as it was redundant.
The build instructions were adapted from those defined earlier in
build.sh. This function has been verfied to work with the supported
target board and supports building without a mount point.
Create mock_clean and mcuboot_clean; the latter is still a dummy
function. The former has been implemented and verified to clean the
target build directory and dependencies directories.
The build instructions for the requirements installation were adapted
from those defined earlier in build.sh, albeit with minor changes.
These instructions go upto the point where the original virtual
environment is restored. The last step would involve some input from the
user to modify the mbed_app.json file in the application folder.
This is quite a significant commit - the script now prompts the user on
whether it should auto-update the application/mbed_app.json file; this
serves well for testing purposes where the "yes" would be piped into the
script. Also, part of the script was adapted from the prior
implementation in build.sh.

Note that this introduces a new dependency "jq" that's not installed
through pip - rather, it requires some manual intervention on the
end-user's behalf.
These steps were adapted from the ones written earlier in build.sh.
The trap invokes clean upon unsuccessful termination.
If there are a number of build files to clean, there is quite a
significant pause from when an error occurs to when control is handed
over to the user. This message indicates to the end-user what's going on
during that time.
Please refer to the comments in the .yml file for more information
Fix dependency issue in workflow
The workflow is no longer triggered on a push to main. This was done to avoid duplicate runs when a pull request is merged.

# This function would clean up the example builds and generated files
clean () {
say message "Cleaning builds and generated files..."
Copy link
Member

@paul-szczepanek-arm paul-szczepanek-arm Aug 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this? nevermind, figured it out

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tool accepts a -c or --clean option to remove any generated build folders or files. It removes the venv folder and any example-specific generated folders and files by invoking the clean functions of each example - mock_clean and mcuboot_clean; the definitions of these functions are located in mock.sh and mcuboot.sh respectively. Also, it's worth mentioning that if the build were to fail at any point with an exit code of 1, it would trap into the cleanup routine, which would call this function.

@sudotensor
Copy link
Contributor Author

sudotensor commented Aug 5, 2021

This pull request will be closed in favour of a new one (#3), which involves changes to the commit history. Please refer to the remarks at the end of the new pull request's description for more information.

Request closed 🛑

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants